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Abstract. Compressed sensing is a framework that makes it possible to recover 
an TV-dimensional sparse vector x E M. N from its linear transformation y <= R M 
of lower dimensionality M < N. A scheme further reducing the data size of the 
compressed expression by using only the sign of each entry of y to recover x was 
recently proposed. This is often termed the 1-bit compressed sensing. Here we analyze 
the typical performance of an Zi-norm based signal recovery scheme for the 1-bit 
compressed sensing using statistical mechanics methods. We show that the signal 
recovery performance predicted by the replica method under the replica symmetric 
ansatz, which turns out to be locally unstable for modes breaking the replica symmetry, 
is in a good consistency with experimental results of an approximate recovery algorithm 
developed earlier. This suggests that the Zi-based recovery problem typically has many 
local optima of a similar recovery accuracy, which can be achieved by the approximate 
algorithm. We also develop another approximate recovery algorithm inspired by the 
cavity method. Numerical experiments show that when the density of nonzero entries 
in the original signal is relatively large the new algorithm offers better performance 
than the abovementioned scheme and does so with a lower computational cost. 
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1. Introduction 

Compressed (or compressive) sensing (CS) is a technique for recovering a high- 
dimensional signal from lower-dimensional data, whose components represent partial 
information about the signal, by utilizing prior knowledge on the sparsity of the signal 
PP. The research field of CS is one of the main topics in information science nowadays 
and has been intensively investigated from the theoretical point of view [2j [3j HJ E] . 
This technique has also been used in various engineering fields pp. 

Let us suppose a situation that an N- dimensional vector x° is linearly transformed 
into an M-dimensional vector y by an M x N measurement matrix $, where y = <&x°. 
The signal is recovered from y by determining the sparsest signal that is consistent with 
the measurements. When M < N, the measurement usually loses some information 
and the inverse problem has an infinite number of solutions. However, when the 
iV-dimensional signal is guaranteed to have only K < M nonzero entries in some 
convenient basis and the measurement matrix is incoherent with that basis, there is a 
high probability that the inverse problem has a unique and exact solution. For example, 
smooth signals and piecewise-smooth signals, like natural images or communications 
signals, typically have a representation in a sparsity-inducing basis such as a Fourier or 
wavelet basis [H |9] . 

Although most of the compressed sensing literature has not explicitly handled 
quantization of the measured data until recently, quantizing continuous data 
is unavoidable in most real-world applications, particularly those in which the 
measurement is accompanied by digital information transmission [10]. Addressing 
the practical relevance of CS in such operation, Boufounos and Baraniuk recently 
proposed and examined a CS scheme, often called 1-bit compressed sensing (1-bit CS), 
in which the signal is recovered from only the sign data of the linear measurements 
y = sign (<fra; ), where sign(x) = x/\x\ for i / operates for vectors in the component- 
wise manner [11]. Although in 1-bit CS the amplitude information is lost during the 
measurement stage, making perfect recovery of the original signal impossible, discarding 
the amplitude information can significantly reduce the amount of data that needs to 
be stored and/or transmitted. This is highly advantageous when perfect recovery is 
not required. In addition, quantization to the 1-bit (sign) information is appealing in 
hardware implementations because 1-bit quantizer takes the form of a comparator to 
zero and does not suffer from dynamic range issues. The scheme is considered practical 
relevant in situations where measurements are inexpensive and precise quantization is 
expensive, in which the cost of measurements should be quantified by the number of 
total bits needed to store the data instead of by the number of measurements. 

The purpose of this paper is to explore the abilities and limitations of a 1-bit CS 
scheme utilizing statistical mechanics methods. In [H] an approximate signal recovery 
algorithm based on minimization of the /x-norm ||£e||i = X^=i \ x i\ un der the constraint 
of sign (<&x) = y was proposed and its utility was shown by numerical experiments. 
Quantization to the sign information, however, leads to the loss of the convexity of 
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the resulting optimization problem, which makes it difficult to mathematically examine 
how well the obtained solution approximates the correct solution. Comparing (in terms 
of the mean square error) the results of numerical experiments with the theoretical 
prediction evaluated by the replica method [12J, we will show that the performance 
of the approximate algorithm is nearly as good as that potentially achievable by the 
Zi-based scheme. We will also develop another approximate algorithm inspired by the 
cavity method [131 EH] an d will show that when the density of nonzero entries of the 
original signal is relatively high the new algorithm offers better recovery performance 
with much lower computational cost. 

This paper is organized as follows. The next section sets up the problem that we 
will focus on when explaining the 1-bit CS scheme. Section 3 uses the replica method 
to examine the signal recovery performance achieved by the scheme. In section 4 an 
approximate signal recovery algorithm based on the cavity method is developed and 
evaluated, and the final section is devoted to a summary. 

2. Problem setup 

Let us suppose that entry x° (i = 1, 2, . . . , N) of iV-dimensional signal (vector) x° G M. N 
is independently generated from an identical sparse distribution: 

P(x) = {l-p)6{x)+pP{x), (1) 

where p G [0, 1] represents the density of nonzero entries in the signal and P(x) is a 
distribution function of x G R that does not have finite mass at x — 0. In 1-bit CS the 
measurement is performed as 

y = sign (®x°) , (2) 

where for simplicity we assume that each entry of M x N measurement matrix $ is 
provided as an independent sample from an identical Gaussian distribution of zero mean 
and variance N~ x . 

Given y and the signal reconstruction is carried out by searching for a sparse 
vector x = (xi) G R under the constraint of sign ($cc) = y. For this task the authors 
of [11] proposed a scheme of 

min { 1 1 a? 1 1 1 } subj. to sign (<&£c) = y and | \x\ | 2 = V~N, (3) 

X 

based on the /i-recovery method widely used and studied for standard CS problems 

p. Here ||a?||i = an d \\ x \\2 — \ x \ — x 1 denote the l\- and Z 2 -no rm s 

of x, respectively. The measurement process of ([2]) completely erases the information 
of length I x° | , which makes it impossible to recover the signal uniquely. We therefore 
introduce an extra normalization constraint \x\ — y/~N for the recovered signal x, and we 
consider the recovery successful when the direction cosine x° ■ x/(\x°\ \x\) is sufficiently 
large. 

Unlike the standard CS problem, finding a solution of ()3]) is non-trivial because 
the norm constraint \x\ = keeps it from being a convex optimization problem 
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Figure 1. Graphical representations of (a) standard and (b) 1-bit CS problems in the 
case of N = 2, M = 1, and K = pN = 1. (a): A thick line and a square of thin lines 
represent a measurement result y = $1X1 + $2 and a contour of Zi-norm \xi\ + \x2\, 
respectively. The optimal solution denoted by a circle is uniquely determined since both 
the set of feasible solutions y — <&iX\ + $2 £2 and the cost function |#i| + |iE2| are convex, 
(b): The shaded area y x ($1X1 + > represents the region that is compatible 

with the sign information of the linear measurement y — $>ixi + $2^2 (dotted broken 
line) . This and the ^-norm constraint x\ + x\ = 2 yield the set of feasible solutions as 
a semicircle (thick curve), which is not a convex set. As a consequence, the constraint 
optimization problem of © generally has multiple solutions (two circles). 

(Figures [T] (a) and (b)). The authors of [TTJ also developed, as a practically feasible 
solution, a double-loop algorithm called Renormalized Fixed Point Iteration (RFPI) 
that combines a gradient descent method and enforcement to a sphere of a fixed radius. 
It is summarized in Figure [21 

The practical utility of RFPI was shown by numerical experiments, but how good 
solutions are actually obtained is unclear because in general the algorithm can be 
trapped at various local optima. One of our main concerns is therefore to theoretically 
evaluate the typical performance of the global minimum solution of ([3]) for examining 
the possibility of performance improvement. 

3. Performance assessment by the replica method 

The partition function 




where (x) = 1 and for x > and x < 0, respectively, offers the basis for our analysis. 
As /3 tends to infinity, the integral of (Hj) is dominated by the correct solution of (JB]). One 
therefore can evaluate the performance of the solution by examining the macroscopic 
behavior of equation (111) in the limit of j3 — > 00. 
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Algorithm 1: Renormalized Fixed Point Iteration (5, A) 



1) Initialization : 

Seed : x s -t 1 1 x 1 1 2 = VN, 

Descent step size : 5 
Counter : k <— 

2) Counter Increase : 

k <- k + 1 

3) One-sided quadratic gradient : 

f fc <r- (Y$) T / / (Y$x/ c _i) 

4) Gradient projection on sphere surface : 

ffc <- f fe - (f fe ,x fc _i)x fe _i/A^ 

5) One-sided quadratic gradient descent : 

h <- x fc _i - Sf k 

6) Shrinkage (/x-gradient descent) : 

(u)j <- sign((h)j)max{|(h)j| - f , 0} for all z 

7) Normalization : 

8) Iteration : Repeat from 2) until convergence. 



Figure 2. Pseudocode for the inner loop of the Renormalized Fixed Point Iteration 
(RFPI) proposed in [TT]. The function f'{x) in step 3 is defined as f'(x) = x for x < 
and 0, otherwise, and it operates on a vector in a component-wise manner. In the 
original expression in [11] the normalization constraint is introduced as 1 1 5c/- 1 ] 2 = 1, 
but we here use ||x/c||2 = V^V for convenience in considering the large system limit 
of N — >• 00. RFPI is a double- loop algorithm. In the outer loop the parameter A is 
increased as A„ = cA„_x, where c > 1 and n are a certain constant and the counter 
of the outer loop, respectively. The convergent solution of i — 1th outer loop is used 
for the initial state of the inner loop of the ith outer loop. The algorithm terminates 
when difference between the convergent solutions of two successive outer loops become 
sufficiently small. 

A characteristic feature of the current problem is that (j4j) depends on the 
predetermined random variables $ and x°, which requires us to assess the average of 
free energy density / = — (/JiV) -1 [hxZ(/3; $,£C°)]^ a , when evaluating the performance 
for typical samples of $ and x°. Here, [•••]$ 3.0 denotes the configurational average 
concerning $ and x°. Because directly averaging the logarithm of the partition function 
is technically difficult, we here resort to the replica method [12J. 

For this we first evaluate n-th moment of the partition function [Z n (/?; <fr, x°)]$, x o 
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for n = 1, 2, . . . G N, using the formula 



/n 
Y[(dx a 5 (\x a \ 2 - N) 

a=l 



x e" 







n M 



0\ 



xnn @ (( $ 

a=l /i=l 

. G N. Here, £c a 



(5) 



which holds only for n = 1,2,... G N. Here, x a (a = 1,2, ...,n) denotes a- 
th replicated signal. Averaging (JSJ) with respect to $ and x results in the saddle 
point evaluation concerning macroscopic variables qo a = q a o = N~ 1 x° ■ x a and 



q a 



Qba 



= N- l x a ■ 



X 



fa, b 



0, 1, 2, . . . , n). Although (j5]) holds only for 



n G N, the expression of iV _1 In [Z n (/?; a; )]^ x0 obtained by the saddle point 
evaluation under a certain assumption concerning the permutation symmetry with 
respect to the replica indices a, b = 1,2, ...n is obtained as an analytic function of 
n, which is likely to also hold for n G R. Therefore, we next utilize the analytic 
function for evaluating the average of the logarithm of the partition function as 
N- 1 In [In Z(P; x°)]^ x0 lim n ^ N' 1 In [Z n x )]^ ^. 

In particular, under the replica symmetric (RS) ansatz where the dominant saddle 
point is assumed to be of the form of 

f p (a = b = 0) 

m (a = 1,2, . . ,,n; b = 0) , . 

1 (a = 6 = 1,2,..., n) ' lJ 

g (a ^b = 1,2,..., n) 

when the distribution of nonzero entries in (CQ) is given as the standard Gaussian 
P(x) = exp(— x 2 /2)/a/27t, the above procedure offers an expression of the average free 
energy density as 



Qba 



f = extr 



1a 1. 




(7) 



in the limit of /3 — > oo. Here a = M/N, extrx{g{X)} denotes extremization of a 
function g(X) with respect to X, u — {x, r m,Q,q,ih}, Dz = dzexp(—z 2 /2)/\^2ir is a 
Gaussian measure, and 



qz + mx°; Q 



1 



qz + mx 




qz + mx \ x + \x 



qz + mx 



(8) 



The derivation of ([7j) is provided in Appendix A 
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The extremization problem of ([7]) yields the following saddle point equations: 



rn 



Q 2 



X = A 




(9) 
(10) 



(11) 



(12) 



(13) 



where H(x) = J °° D2. The value of m determined by these equations physically means 
the typical overlap iV -1 [x° ■ x]^ x0 between the original signal x° and the solution x 
of (131). Therefore the typical value of the direction cosine between x° and x, which 
serves as a performance measure of the current recovery problem, is evaluated as 
[(x° ■ x)/\x°\\x\]^ = Nm/(y/Np x yN) = m/^/p. Alternatively, we may also use as 
a performance measure the mean square error (MSE) between the normalized vectors: 



MSE 



x 

\x\ 



X 



ni 



(14) 



We solved the saddle point equations for various sets of a and p. The curves 
in figures [3] (a)-(d) show the theoretical prediction of MSE evaluated by (fT4l) plotted 
against the measurement bit ratio a = M/N for p = 1/32,1/16,1/8, and 1/4. To 
examine the validity of the RS ansatz, we also evaluated the local stability of the RS 
solutions against the disturbances that break the replica symmetry [T7J, which offers 



a 



<Qxf 



arctan 



x2 \{l-p)H 




1 



1 < 0, 



(15) 



' q + m 2 

as the stability condition. A brief sketch of the derivation of this condition is shown 
in Appendix B Unfortunately, (fl5|) is not satisfied for any regions in figures [3] (a)-(d). 
This is presumably because the optimization problem for ([3]) has many local optima 



reflecting the fact that the constraint of \\x\\ 2 = \N loses the convexity. This indicates 
that taking the replica symmetry breaking (RSB) into account is necessary for evaluating 
the exact performance of the signal recovery scheme defined by Q. 

We nonetheless think that the RS analysis offers considerably accurate 
approximates of the exact performance in terms of MSE. The (x) symbols in figures 
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Figure 3. MSE versus the measurement bit ratio a for the signal recovery scheme 
using ([3]). (a), (b), (c), and (d) correspond to the p — 1/32,1/16,1/8, and 1/4 cases, 
respectively. Curves represent the theoretical prediction evaluated by the RS solution, 
which is locally unstable for disturbances that break the replica symmetry for all 
regions of (a)-(d). Each symbol (x) stands for the experimental estimate obtained for 
RFPI in pj] from 1000 experiments with N = 128 systems. 



[3] (a)-(d) stand for MSE experimentally achieved by RFPI, which were assessed as the 
arithmetic averages over 1000 samples for each condition of N = 128 systems. Excellent 
consistency between the curves and symbols suggests that even if ([3]) has many local 
optima, they are close to one another in terms of the Z 2 -norm yielding similar values of 
MSE. This also implies that RFPI, which is guaranteed to find one of the local optima, 
performs nearly saturates as well (as measured by the MSE) as the signal recovery 
scheme based on (J3]). 

Of course, we have to keep in mind that the consistency between the theory 
and experiments depends highly on the performance measure used. Figures 0] (a)- 
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(a) p = 1/32 
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(b)p=l/16 
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(c) p= 1/8 
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Figure 4. FP and FN probabilities versus the measurement bit ratio a — M/N. (a), 
(b), (c), and (d) corresponds to the p = 1/32,1/16,1/8, and 1/4 cases, respectively. 
Solid and dashed curves represent theoretical predictions obtained by the RS solution 
for FP and FN, respectively. Asterisks and squares denote experimental results for 
FP and FN, respectively. The experimental results were obtained by RFPI from 1000 
samples for each condition of N = 128 systems. 



(d) show the probabilities of wrongly predicting sites of nonzero and zero entries, which 
are sometimes referred to as false positive (FP) and false negative (FN), respectively. 
These indicate that there are considerably large discrepancies between the theory 
and experiments in terms of these performance measures, which is probably due 
to the influence of RSB. Nevertheless, the RS-based theoretical predictions are still 
qualitatively consistent with the experimental results in the way that the probability of 
a FP remains finite even when the measurement bit ratio a = M/N tends to infinity for 
any values of p. This implies that the /i-based scheme is intrinsically unable to correctly 
identify sites of nonzero and zero entries. 
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4. Cavity-inspired signal recovery algorithm 

The analysis so far indicates that the performance of RFPI is good enough in the 
sense that there is little room for improvement in achievable MSE. RFPI requires 
tuning of two parameters 5 and A, however, which is rather laborious. In addition, 
the convergence of the inner loop of Figure [2] is relatively slow, which may limit its 
application range to systems of relatively small sizes. We therefore developed another 
recovery algorithm following the framework of the cavity method of statistical mechanics 
[131 EH] 5 or equivalently, the belief propagation of probabilistic inference [T5l [TB] . 

For simplicity of notations, let us first convert all the measurement results to +1 
by multiplying (fi = 1,2,..., N) to each row of the measurement matrix $ = 
as — > (y^&^i), and newly denote the resultant matrix as $ = (3?^). In the new 
notation, introduction of Lagrange multipliers a = (a M ) and surplus variables z = (z^) 
converts ([3]) to an unconstrained optimization problem: 

™ ^ax J n + it a » ( S ®» iXi - z A + ^\ ll x2 i- N 

L i=l /i=l \i=l J 

N , a x M 



mm max 

x,z>0 a,A 



where z > means that each entry of z is restricted to be positive. 

Coupling terms ■ Q^a^Xi make the optimization of (fl6l) a nontrivial problem. In 
statistical mechanics, a standard approach to resolving such a difficulty is to approximate 
( fT6l) with a bunch of optimizations for single-body cost functions parameterized as 

Ci{xi) = -^-x\ - HiXi + \xi\, (17) 



and 



2 



where Ai,B^,Hi, and Zf^ are parameters to be determined in a self-consistent manner. 

In the cavity method this is done by introducing virtual systems that are defined 
by removing a single variable x» or a single pair of variables (a M , z^) from the original 
system [13J. When A" is sufficiently large, the law of large numbers allows us to assume 
that the values of Aj and are constant independently of their indices; that is, that A 
and B are constants. Under this simplification, this method yields a set of self-consistent 
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equations: 



A? 



= J^^vx, - Ba^ (19) 
«m = " \f (K») , (20) 

M 

Hi = ^$ Mi a^ + rxi, (21) 

At=l 

^ =^'(^), (22) 

where // = 1,2,...,M, i = 1,2,..., N, f(u) = {u 2 /2)Q(-u), and g(u) = 
{{\u\ - l) 2 / 2 ) 6 (\u\ - 1). A is determined so that J2?=i%i = N holds ' and B is 
provided as B = A' 1 (V 1 Y,tx f (Hi)) = A" 1 (a^ 1 £ii 6 (|^| - 1)) . T = 

fi- 1 (iV- 1 Ejli / ff = 5- 1 (iV- 1 EjLi © (1^1 - 1)) , and Txi on the right-hand 

side of ( 12 ip is often referred to as the Onsager reaction term [T8|IT9"]. Equation f )22|) offers 



the recovered signal. The derivations of these equations are provided in |Appendix C 



A distinctive feature of the above set of equations is that they are free from tuning 
parameters such as A and 5 in RFPI, which is highly beneficial in practical use. It 
is therefore unfortunate that in most cases the naive iterations of ffT9l - fT22l hardly 
converge, which is considered a consequence of RSB [20J. 

We found, however, that modifying ( 1T91 and fl2"U|) to 

N 

= a M - -^f | Q^Xi \ , (23) 




in conjunction with handling B as a parameter to be controlled in the outer loop, 
results in a fairly good approximate signal recovery algorithm. The resultant algorithm 
is somewhat similar to RFPI as the combination of (121!) and (1231) roughly acts as the 
One-sided quadratic gradient descent step in Figure [2J However, as the length 
of H = (Hi) = <fr T a is not restricted to a fixed value, the current algorithm does not 
need a small step size S for the convergence. Another significant difference from RFPI 
is the existence of the Onsager reaction term in ( )2~1]) . This term effectively cancels 
the self-feedback effects included in Hi of (ED) , and this is expected to accelerate the 
convergence of the algorithm. A pseudocode for the inner loop is summarized in Figure 
® 

The necessity of controlling B in the outer loop, which is essential for having 
good convergence in the inner loop, means that our algorithm still requires one tuning 
parameter. Nonetheless, the reduction in the number of the tuning parameters from 
two to one is considerably advantageous for practical use. In practice, the initial value 
of B should be set so that only a single entry becomes nonzero. This is easily done by 
the Binary Iterative Hard Thresholding algorithm [21], which requires the number of 
nonzero entries as extra prior knowledge. After the initial value is set, B is reduced as 
B n = rS n _i with an appropriate constant < r < 1, where n is the counter of the outer 
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Algorithm 2: Cavity-inspired Signal Recovery(B,x*,H*) 



1) Initialization : 

X seed : x ^~ x* 

H seed : H <- H* 

Counter : k <r- 

2) Counter increase : 

k <- k + 1 

3) One-sided quadratic gradient descent : 

H fc <- H k _ a - B- 1 (Y$) T / / (Y$x fe _ x ) 

4) Assessment of Onsager coefficient : 

r<-(NB)- 1 l T f' (YSxa-O 

5) Self-feedback cancellation : 

H fc ^— H fe + Tx fc _i 

6) Shrinkage (^-gradient descent) : 

(u)j <- sign((H) i )max{|(H) i | - 1,0} for all i 

7) Normalization : 

x fe <- 

ll u lh 

8) Iteration : Repeat from 2) until convergence. 



Figure 5. Pseudocode for the inner loop of the cavity-inspired signal recovery (CISR) 
algorithm, x* and H* are the convergent vectors of x^ and obtained by the previous 
outer loop. The 1 in step 4) is the TV-dimensional vector all entries of which are unity. 
If (u)j = eventually holds for Vz in step 6), B is reduced so that only maxj{|(u)j|} 
becomes nonzero, and the procedure is restarted from step 3). 

loop. The algorithm terminates when the difference between the convergent solutions 
of two successive outer loops is sufficiently small. 

The MSE results obtained in numerical experiments with the cavity-inspired signal 
recovery (CISR) algorithm are shown in Figures [6] (a)-(d). They indicate that except 
in the case in which the nonzero density p of the original signals is significantly low, 
CISR provides MSE values almost equal to or lower than those of RFPI. Figures [7| (a)- 
(d) show the FP and FN probabilities for CISR. The discrepancies from the theoretical 
prediction are not unexpected because the modification of fl20|) to fl23l) means that CISR 
is no longer based on ([3]) or ( jl~6l) . The FN probabilities for CISR are higher than those 
for RFPI, while the FP probabilities are lower. This implies that CISR has a capability 
of yielding sparser signals than RFPI, which is presumably because parameter B of 
CISR is initially set so that only a single entry of x is nonzero while such a tuning is 
not taken into account in RFPI. 

The run times actually required for performing the experiments in a MATLAB® 
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(c) p= 1/8 
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(b)p = l/16 
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(d) P = 1/4 
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Figure 6. MSE versus measurement bit ratio a for the cavity-inspired signal recovery 
(CISR) algorithm. Experimental conditions are the same as in Figures [3] (a)-(d). 



environment for the cases of iV = 128 and M = 3N = 384 are listed in Table [TJ 
Although the run times of RFPI may be reduced by optimally tuning the descent step 
size 5, CISR is several hundreds of times faster than RFPI. This shows the significant 
computational efficiency of CISR. The NORT values in Table [U are the run times when 
the Onsager reaction term in ( I2T1) was removed from CISR. Their being 1.13-2.37 times 
longer than those for CISR indicates that the cancellation of the self-feedback effects by 
adding the Onsager reaction term speeds the convergence of CISR significantly. 

5. Summary 

In summary, we have examined typical properties of 1-bit compresses sensing (CS) 
proposed in [11] utilizing methods of statistical mechanics. Signal recovery based on 
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(c) p= 1/8 
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Figure 7. FP and FN probabilities of versus measurement bit ratio a for the CISR. 
Experimental conditions are the same as in Figures [4] (a)-(d). 



the Zx-norm minimization is a standard approach in CS research. Unlike the normal 
CS scheme, however, the Zi-based signal recovery cannot be formulated as a convex 
optimization problem, which makes practically performing it nontrivial. 

We have shown that the theoretical prediction of the performance of the /i-based 
scheme, which is obtained by the replica method under the replica symmetric (RS) 
ansatz, exhibits a fairly good accordance (in terms of MSE) with experimental results 
obtained using for an approximate signal recovery algorithm, RFPI, proposed in [TT] . 
The replica symmetry of the RS solution turned out to be broken, however, which implies 
that there are many local optima for the optimization problem of the signal recovery. 
Our results suggest that the local optima, which can be searched by RFPI, yield similar 
values of MSE representing the potential performance limit of /i-based recovery scheme. 

We have also developed an approximate signal recovery algorithm utilizing the 
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Table 1. Comparison of computational costs for the N = 128 and M = 3N = 384 
cases. The values listed here are the average run times (in seconds) evaluated in 1000 
experiments, and the numbers in parentheses are the standard deviations. In RFPI, 
5 was roughly tuned as 0.01, and A was enlarged as A n = 2A„_i with the initial 
value Ao = 0.005 in the outer loop. On the other hand, B of CISR was reduced as 
B n = 0.9-B„_i. The NORT values are the run times required for performing the same 
experiments when the Onsager reaction term was removed from CISR. In all cases the 
algorithms terminated when the difference per entry, in terms of Zi-norm, between the 
convergent solutions of two successive outer loops was less than 10~ 8 . 





K = 4 


K = 8 


K= 16 


K = 32 


RFPI 
CISR 
NORT 


25.7636(10.0799)s 

0.0385(0.0583)s 

0.0557(0.0889)s 


27.8293(3.3566)s 

0.0705(0.1058)s 

0.0795(0.1095)s 


33.3552(3.2914)s 

0.0245(0.0346)s 

0.0581(0.0566)s 


35.4574(3.3869)s 
0.0247(0.0207) s 
0.0369(0.0316)s 



cavity method. Naive iterations of self-consistent equations derived directly from the 
cavity method hardly converge in most cases, which can be regarded as a consequence 
of the replica symmetry breaking. However, we have shown that modification of one 
equation in an appropriate manner, in conjunction with controlling a macroscopic 
variable in the outer loop, results in a fairly good signal recovery algorithm. Compared 
with RFPI, the resultant algorithm is beneficial in that the number of tuning parameters 
is reduced from two to one. Numerical experiments have also shown that whenever the 
density of nonzero entries of the original signal is not considerably small the cavity- 
inspired algorithm performs as well as or better than RFPI (in terms of MSE) and has 
a lower computational cost. 

We here focused on the /i-based recovery scheme since it was proposed and 
examined in the seminal paper on 1-bit CS [11] . However, the significance of the Zi-based 
scheme may be rather weak for 1-bit CS because the loss of convexity it entails keeps it 
from leading to the development of mathematically guaranteed and practically feasible 
algorithms. Therefore, much effort should be devoted to developing recovery algorithms 
following various principles. For example, the idea based on the Bayesian inference and 
matrix design that was proposed for standard CS [6] may also be a promising approach 
for 1-bit CS. 

Acknowledgments 

YX acknowledges a scholarship from Rotary Yoneyama Memorial Foundation, Inc. This 
study was partially supported by JSPS KAKENHI Nos. 22300003 and 22300098 (YK). 



Statistical mechanics approach to 1-bit compressed sensing 
Appendix A. Derivation of (T7|) 



16 



Appendix A.l. Assessment of [Z n $, £C°)]$ x0 for n e N 

Averaging (JSJ) with respect to $ and a; offers the following expression of the n-th 
moment of the partition function: 



„ n 



x e 



-l3\\ X "-\ 



X 



a=l 

n M 



nn e ((*A*(** a W 

a=l fi=l 



(A.l) 



We insert n(n + l)/2 trivial identities 

1 = JV / rfg afe 5 (x a ■ x b - Nq ab ) , (A. 2) 

where a > b — 0, 1, 2, . . . , n, into (1A.1I) . Furthermore, we define a joint distribution of 
ri + 1 vectors {x a } = {x°, x 1 , x 2 , . . . , x n } as 



P({«°}|Q) 



-P x u ) x 



noKi*°i a -*) 



a=l 



x l[6 (x a ■ x b - Nq ab ) , 

a>b 



X e 



\\x" 



") 



(A.3) 



where Q = (g^) is an (n+1) x (n+1) symmetric matrix whose 00 and the other diagonal 
entries are fixed as p and 1, respectively. P(x°) = YliLi ((1 ~~ P)^{ X T) + pP{ x i)j denotes 
the distribution of the original signal x°, and V (Q) is the normalization constant that 
makes J YYl=o dx a P ({x a }\Q) = 1 hold. These indicate that flA.lj) can also be expressed 
as 



[Z n (P;$,x°)]^ x0 = f dQ(V(Q)x~(Q)) 
where dQ = Yl a>b dq ab and 

/n 
l[dx a P({x a }\Q) 
a=0 



n M 

nne(M,(^ a )j 

o=l fl=l 



(A.4) 



(A.5) 



Equation (1A.5|) can be regarded as the average of YYl=i Yl^=i ® ((*&x°)^(<f>x a )^) 
with respect to {x a } and $ over distributions of P ({x a }) and -P(*fr) = 

\ —MN , \ 

^/2tt/N) exp ( — (N/2) i $^ J . In computing this, it is noteworthy that the 

central limit theorem guarantees that = (<&£E a ) M = Q^iX® can be handled as 

zero-mean multivariate Gaussian random numbers whose variance and covariance are 
provided by 



[ U X]* >{(B « } = <W?« 



(A.6) 
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when $ and {x a } are generated independently from P(<fr) and P({x a }), respectively. 
This means that flA~5]) can be evaluated as 

^^exp(-^ T Q-^)nLi0^ O " a ^ M 



( 2vr )(n+l)/2( det Q)l/2 

dwexp (-f^Q- 1 ™) e («°) nLi K) 



i'li 



( 27r )(n+l)/2( det Q)l/2 



On the other hand, expressions 



1 f +1 °° ( 1 

5 (|a; a | 2 - N) = — J dq aa exp i--q aa (| 



,a|2 



X 



N) 



and 



5 (a; a • - Nq ab ) = — I dq ab exp (q ab (x a ■ x b - Nq ab ) ) 

^ J -\oo 



and use of the saddle point method offer 

llnV(Q) = extr j-^TrQQ 



(A.7) 



(A.8) 



(A.9) 



+ In {^J dxP(x°) exp (^x T Qx - (3 Pl^l) j J ■ ( A - 10 ) 



Here x 



(x°, x 1 , 



X 



n\T 



and Q is an (n + 1) x (n + 1) symmetric matrix whose 



00 and other diagonal components are given as and — q aa , respectively, while 
off-diagonal entries are offered as q ab . Equations ( 1A.7I) and ( lA.lOj) indicate that 
A -1 In [Z n ((3; <fr, x )}^ x0 is correctly evaluated by using the saddle point method with 
respect to Q in the assessment of the right-hand side of ( 1A.4|) when N and M tend to 
infinity keeping a = M/N finite. 



Appendix A. 2. Treatment under the replica symmetric ansatz 



Let us assume that the relevant saddle point in assessing ( 1A.4I) is of the form of (E]) and, 
accordingly, 



0, (a 



0) 



(jab — <jba — < 



m, (a = 1, 2, . . . , n; b = 
<5, (a = 6 = 1,2, . . . ,n) 
<?, (o^6=l,2,...,n) 
o „,i 



0) 



(A.11) 



n+1 dimensional Gaussian random variables u , u , . . . u n whose variance and covariance 
are provided as §6§ can be expressed as 



m 



u 



u" 



P s +—z, 

\/l-qs a + y/qz, (a 



1,2, 



n) 



(A.12) 
(A.13) 
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utilizing n + 2 independent standard Gaussian random variables z and s°, s 1 , . . . , s n . 
This indicates that flA.7j) is evaluated as 

M 

5(Q)=(2/Dzff| = ^ ) H n ( \[-r^—;z) ) ■ (A.14) 



y/ pq — m 2 J VV ^ ~ 1 
On the other hand, substituting (lA.llj) into (lA.lOj) . in conjunction with the identity 



exp J q x a x h 

a>b(>l) 



Dzexp ( ~ 



,a=l 



q (x a ? 



qzx 



(A.15) 



provides 



1 

N 



',q,m 



n(n — 1) 



In V(Q) = extr < — Q — -qq — mm 



+ ln 



dx exp | — ^^~^ x 2 + ^/^z+mx^x — 



(A.16) 



Although we have assumed that n G N, the expressions of (1A.14P and (IA.160 
are likely to hold for n e R as well. Therefore the average free energy 
/ can be evaluated by substituting these expressions into the formula / = 
- Km^oid/dn) ((/3A0" 1 In [Z n (f3; x )}^ . 

In the limit of /3 — > oo, a nontrivial saddle point is obtained only when \ = 0(1 — q) 
is kept finite. Accordingly, we change the notations of the auxiliary variables as 
Q + q — > j3Q, q — > j3 2 q, and m — > j3m. Furthermore, we use the asymptotic forms 



lim \ [ DzH ( m 
/3->oo p J yy/pq — m? 



z\\nH 



l-q 



DzH 



-{ 



m z) -^e(z] 



arctan 



\J p — rn 2 



rn 



— VP - m 

P 



(A.17) 



and 



^lim — In dxexpy3 h- ^-x 2 -\- (^/^z+mx^jx — \x\ 

= -<p [y/^z + mi"; . 
Using these in the resultant expression of / offers ([7j). 



(A.18) 



Appendix B. Stability of the RS solution 



The 1-step replica symmetry breaking (1RSB) ansatz means that, at the relevant saddle 
point, n replica indices 1,2, ... ,n are classified into n/p groups of an equal size p, and 
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lab = 9i holds if a and b belong to an identical group and go(< gi), otherwise. This 
yields the following expression of the average free energy of finite temperature: 



/ = extr { -- 



In 



Bt exp (-py ] 
P 



1 9i 
- 2q{Q + 9i) + - 9i) + fgifofr ~ ?o9o) + -gW/// 



m 



pg — m z 



z In / Dtexp(-p3 ; i, 



(b.i; 



where y = - In ^ f cfeexp (-(Q + qi)x 2 /2 + (y^ - go^ + v^- 2 + mx°)x - 

3>i = -ln(/Dx0 (- (VI - Qix + Vft -9o^ + v^o z )))' u = {Qi>Qo> m >Q>&>Qo,™}, 
and [•••] x o 2 = / dx°P(x°) fDz (■■■). The RS solution is regarded as a special 
case of the 1RSB solution for which qi = go holds. Therefore one can check the 
thermodynamical validity of the RS solution by examining the stability of the solution 
of qi = g under the 1RSB ansatz. 

The extremization condition of ( IB. II) indicates that 



qi - qo 



fpte-py°(dy /d(vi z)y 

f Dte-py° 



jDte- 



and 



d 2 y™ 

d(y/q~ z) 

d 2 y™ 



9i — 9o — 2a / DzH 



J Bte- py °t 2 ( J Bte- py °t 



J Dte~py<> 
(Qi ~ Qo) 



fnte- 




(9i - 9o) 



(B.2) 
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y/pq-m 




jDte-^ {dy 1 /d{^z)) 



fDte- 



2a / DzH 




x 



fBte-P y H 2 ffDte-pyif 



2a / DzH 




(9i - 9q) 



(9i - 9o ) 



(B.3) 



pq — rrr 

hold for \q\ — q \ <C 1 and |gi — go I ^ 1 irrespectively of the value of p. Here y^ s 
and yp s represent assessments of y and 3^1 under the assumptions of gi = g and 
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qi = qo, respectively. In (1B.2P and (1B.3P we used the Taylor expansion expressions 
dy /d(Vtoz) ~ dy^/d(y/^z) + d 2 y* s /d(Vfoz) 2 VgT=%t and dy^d^z) ~ 
dyf^ / d(y/q~oz) + d 2 y^ s / d(y/q~oz) 2 y/qi — q t, and the fact that the variances of t for 
the measures Dte _p ^ / jDte _p ^° and Dte - ^ 1 / jDte"^ 1 become unity as gi — go and 
Qi — % vanish, irrespectively of the value of p. 

To examine the stability of the RS solution in the limit of /3 — > oo, let us change the 
variable notations as x — — q), Q + qi — > 0Q, q\ — > (3 2 qi, qo —> (3 2 qo, and rh —> (3m 
and set q = q and % = q. This yields expressions of y^ s — P4>{\/q z + rhx°; Q) = 
—f3g(y/qz + rhx°)/Q and y^ s ~ (f3/x)f(~\^l z ) f° r /3 > 1. Substituting these into 
f lB~2l) and flB~3l) leads to 



1 

A ~ — 

Q 2 



g"{yjqz + rhx°] 



A 



(B.4) 



and 



A ~ — / D,z# -== 



(n-«)) a A, 



(B.5) 



where we set A = q\ — q and A = q\ — q, and used q — > 1. The condition that (IB. 41) 
and OESJ allow a solution of (A, A) ^ (0, 0) offers (fT5|). 



Appendix C. Derivation of the cavity equations 

We refer to the system in which Xi and (a M , z M ) are kept out as the z-cavity and fi- 
cavity systems, respectively. In addition, we denote Ci^^Xi), and as the 

single-body cost function for the /i-cavity system and its parameters, respectively, and 
similarly for £ M _+j(a M , -B^j and K^i. Self-consistent equations are derived from 
the following arguments. 

Vertical step: 

Let us suppose that Xi is put into the i-cavity system, which yields an approximation 
of the cost function of ( |T6l) as (A/2)xf + \xi\ + z~2^=i {£v-ti(Q>v>Zv) + ^ui^uXi). From 
this function we remove all terms that are related to (a M , z^) of a certain index 
\i G {1, 2, ... , M}, which leads to an approximate cost function of the /i-cavity system. 
Ci^^(xi) must be obtained by partially optimizing the resulting /i-cavity cost function 
with respect to (a u ,z u ) of the remaining indices W G {1, 2, . . . , M}\/i, where 5\a 
generally denotes the set provided by removing an element a from a set S. This offers 
the relation 

Ci-+u{xi) = t;X 2 + + (mm max {£^(^,2^) + ^ ui a u Xi} \ . (C.l) 

This relation and the fact that is a negligibly small independent sample from an 
identical Gaussian distribution with zero mean and variance N~ x yield the following 
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equations evaluating and from a set of {B u ^i} and {K u ^i}: 

B 



=A + J2 -P^fi^il (C2) 



H^,= -J2P^f'( K ^- ( C - 3 ) 

"TP 

Horizontal step: 

Similarly, putting (a^, 2^) into the /x-cavity system and removing Xi yields another 
relation, 



(C.4) 



which offers 



^=Er L j'W' ( c - 6 ) 



• -^j->," 



Recovery step: 

and ifj are evaluated from (10.51) and (1C.6P as 



1 — u—*t 

fj,=l I* 



M 

B 



|x=l ^ 

This means that the recovered signal is provided as 

x i =±-g'(H i ), (C.9) 

where A is determined in such a way that YliLi ^ = N holds. Similarly, 

i=i 

AT , 

i=l 

are obtained from ( 1C2[) and ( 1C.3I) . These offer the (approximate) optimal value of the 
Lagrange multiplier 

a, = ~f(K^). (C.12) 

Equations (1C3|) and (1C.8j) indicate the difference between and ifj is 

vanishingly small for iV — > oo as scales as O (iV -1 / 2 ), and similarly for K^-^ and 



Statistical mechanics approach to 1-bit compressed sensing 



22 



K^. This also allows us to handle A4 and as a single site-independent parameter 
A, and we similarly deal with and -B m _h as B. These considerations, in conjunction 
with (EZD and (ICTlOl) . offer 

M 

^ = A + atr£/'W> (C13) 



B 



1 - 



iVv4 



(C.14) 



i=l 



where we replaced $^ in ( 1C.7I) and (IC.10I) with its expectation iV 1 by utilizing 
the law of large numbers. Furthermore, inserting f'(K^i) ~ f'{K^ — Q^Xi) ~ 
- $ M r(^)^ and (/(tf^) ~ + ~ cf(Hi) + $^"(#0^ into 

(1(180 and (IC.lip . respectively, yields 



M / 1 M \ 

M / M 



.1 7 



(C.15) 



and 



A' 



1=1 

N 

~ E $ 

i=l 
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lE*^"W 

i=l 



(C.16) 



t=i 



where we set T = (NS) -1 EjLi Equations (109]) . ( 1EH2|) . and flCT3|) - flaT6|) 
lead to (frg])-(F221). 



References 

[1] Candes E J and Wakin M B, 2008 IEEE Signal Processing Magazine March 2008, 21 
[2] Donoho D L, 2006 IEEE Trans. Inform. Theory 52 1289 

[3] Candes E J, Romberg J and Tao T, 2006 IEEE Trans. Inform. Theory 52 489 
[4] Kabashima Y, Wadayama T and Tanaka T, 2009 J. Stat. Mech. (2009) L09003; J. Stat. Mech. 
(2012) E07001 

[5] Ganguli S and Sompolinsky H, 2010 Phys. Rev. Lett. 104 188701 

[6] Krzakala F, Mezard M, Sausset F, Sun Y F and Zdeborova L 2012 Phys. Rev. X 2 021005 
[7] https : / /sites .google . com/ site/ igorcarron2/compressedsensinghardware 
[8] Elad M, 2010 Sparse and Redundant Representations: From Theory to Applications in Signal and 
Image Processing (New York: Springer) 



Statistical mechanics approach to 1-bit compressed sensing 



23 



[9] Starck J-L, Murtagh F and Fadili J M, 2010 Sparse Image and Signal Processing: Wavelets, 
Curvelets, Morphological Diversity (New York: Cambridge University Press) 
[10] Lee D, Sasaki T, Yamada T, Akabane K, Yamaguchi Y and Uehara K, 2012 in Proceedings of 

IEEE Vehicular Technology Conference (VTC Spring) 
[11] Boufounos P T and Baraniuk R G 2008 m Proceedings of CISS2008 16 

[12] Dotsenko V S, 2001 Introduction to the replica Theory of Disordered Statistical Systems, 

(Cambridge: Cambridge University Press) 
[13] Mezard M, Parisi G and Virasoro M A, 1987 Spin Glass Theory and Beyond (Singapore: World 

Scientific) 

[14] Mezard M and Montanari M, 2009 Information, Physics, and Computation (New York: Oxford 
University Press) 

[15] MacKay D J C 1999 IEEE Trans. Inform. Theory 45 399; MacKay D J C and Neal R M, 1997 

Elect. Lett. 33 457 
[16] Kabashima Y and Saad D, 1998 Europhys Lett 44 668 
[17] de Almeida J R L and Thouless D J, 1978 J. Phys. A 11 983 
[18] Thouless D J, Anderson P W and Palmer R G, 1977 Phil. Mag. 35 593 
[19] Shiino M and Fukai T, 1992 J. Phys. A 25 L375 
[20] Kabashima Y, 2003 J. Phys. A 36 11111 

[21] Jacques L, Laska J N and Boufounos P T and Baranuik R G, 2011 Robust 1-Bit Compressive 
Sensing via Binary Stable Embeddings of Sparse Vectors arXiv:1104.3160 



